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Issues in Program Evaluation Research 
in State Rehabilitation Agencies 



Evaluation research is becoming increasingly sig- 
nificant to the vocational rehabilitation field with 
the current emphasis on program evaluation and account- 
ability. The series of papers abstracted here were 
presented as a Symposium at the American Personnel and 
Guidance Association Convention in New Orleans, Louisiana 
April 11, 1974, and review several key issues included 
in evaluation research. 



A large proportion of the research conducted within 
State Vocational Rehabilitation Agencies during the next 
few years will probably be evaluative in nature and oriented 
toward an immediate utilization objective. With this 
research direction in mind three basic models of evalua- 
tive research are introduced: Model 1 - measures the 
effect of an entire program; Model 2 - compares the specific 
variables within a program; and Model 3 • measures the 
differential effects of specific variables within a program-. 
Model 3 is considered to have the greatest utility for 
providing data on which state rehabilitation agency admin- 
istrators can make effective program decisions. The 
reasons for such a conclusion are put forth. Model 3 is 
then comprehensively described. The importance of practi- . 
tioner involvement in the development of research questions 
and plans is discussed. 
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Designing State Vocational 
Rehabilitation Agency Evaluation Research 
Stanford E. Rubin 



ABSTRACT 

Problems in Measuring Outcomes in 
Rehabilitation Research Projects 
Brian Bolton 

This paper consists of six parts. The first presents 
a brief introduction to the criterion problem in rehabili- 
tation counseling ♦ The second outlines four conceptual 
issues relevant to the measurement of outcomes (economic 
vc* noneconomic, unidimensional vs. multidimensional, 
absolute vs. relative, and self -report vs. observer ratings) • 
The third reviews seven measures of client outcome which 
have been developed to address the criterion problem. The 
fourth briefly discusses two standard procedures for quan- 
tifying psychometric change (raw score and residual score 
methods) . The fifth reviews three studies which support 
the conclusion that rehabilitation outcome is a multidimen- 
sional construct. The final section discusses the role of 
factor analysis in the development of models ^of the reha- 
bilitation process and outlines two standard models. 



ABSTRACT 

Assessment of External Effects 
in Rehabilitation Research 
Reed Greenwood 

„ .Eyaluative research in rehabilitation calls for 
approaches which recognize that rehabilitation outcome 
is determined by a complex group of factors, both in- 
ternal and external to the rehabilitation system. Fo- 
cusing on external or environmental influences affect- 
ing rehabilitation outcome, this presentation reviews 
the need for expanding the parameters of evaluative 
research to a system approved at both mezzo (inter- 
mediate) and macrosystem levels. Specific external 
factors at mezzo and macro levels include social climate, 
geography, social mores and folkways, economic conditions 
and institutional policies and practices. 



ABSTRACT 

Issues in Initiating Large 
Scale Rehabilitation Research 
Richard Roessler 

Problems encountered in conducting large scale 
evaluation studies in rehabilitation settings are discussed 
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in the context of a Services Coordination Project 
(Arkansas Services Center, Jonesboro, Arkansas) . In part, 
the paper discusses differences in perspectives heid by 
those involved in different phases of the project; e.g., 
those responsible for initiating and managing the pro- 
ject andJLhose responsible for evaluating its effective- 
ness. Other issues such as setting objectives, defining 
the experimental treatment, etc. are included. Finally, 
an effort is made to present resolutions to problems 
identified in the paper. 



ABSTRACT 

Reactions on Evaluation 
Jerold D. Bozarth 

Research and evaluation are often assumed to be 
different types of processes* Essentially, good eval- 
uation is the same as good research since both pro- 
cesses require characteristics such as internal and 
external validity. However, most evaluation is at the 
lower end of the qualitative continuum and may be 
quite harmful if results from such evaluations are used 
to determine policy, or to modify programs. Evaluation 
research should be relevant to questions posed by groups 
such as legislators and boards of supervisors, and should 
be directed toward program modification. These questions 
must often be restated in ways which allow measurement, 
other suggestions offered to strengthen evaluation research 
are to utilize a "helping" set rather than the more 
threatening "accountability" approach and to incorporate 
basic research into program evaluation efforts. 

A copy of this publication may be obtained by writing to; 
Publications Section 

Arkansas Rehabilitation and Training Center 
Hot Springs, Arkansas 71901 
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in State Rehabilitation Agencies 
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Arkansas Rehabilitation Research and Training Center are 
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for program participation and/or services have a right to 
file complaints and to appeal according to regulations 
governing this principle • 
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« INTRODUCTION 

♦ The Arkansas Rehabilitation Research and Training 
Center publishes papers which relate to topics of current 
significance to the field of vocational rehabilitation. 
Recognizing the importance of evaluative research in 
current program evaluation efforts being conducted in 
publifc vocational rehabilitation agencies, the authors 
prepared the papers included in this publication. Although 
not exhaustive in content, the papers should give the 
reader a feel for the complexities involved in evaluative 
research, by explicating design and measurement con- 
siderations, and additionally identifying environmental 
constraints and practical issues involved in implementing 
such research. The authors invite feedback and reactions 
to the presentations. 

Reed Greenwood, Ed.D. 
Director of Research 
Arkansas Rehabilitation 
Research and Training Center 
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DESIGNING STATE VOCATIONAL JREHABILITATION 
AGENCY EVALUATION RESEARCH* 

Stanford -E. Rubin, Ed.D. 
Arkansas Rehabilitation Research and Training Center 
University of Arkansas 



Since we are living through the decade of accounta- 
bility, a large proportion of the research conducted with- 
in State Vocational Rehabilitation Agencies during the 
next few years will probably be evaluative in nature. 
Such research will be aimed at providing data on which 
^ program or sub-program termination, selective utilization, 
modification, or expansion decisions can be based. There- 
fore, such research will ob^'iously be oriented" toward an 
immediate utilization objective. Suchman (1967) succinct- 
ly reinforces this p*oint with the following s-tatement: 



"Unlike the basic researcher, the applied re- 
searcher must be constantly aware of the 
potential utility of his findings. Only rarely 
can he take consolation in the fact that the 
operation was a success but the patient died . '* (p. 21) 

In spite of its applied nature, though, evaluative 
research must still adhere as closely as possible to the 
same rules governing basic research if any faith is to be 
placed in its findings. For example, evaluative research 



*This paper is based on a presentation at the American Per- 
sonnel and Guidance Association Convention in New Orleans, 
April 11, 1974. 
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must utilize measures of program activity and program 
effect that have ^^demonst rated reliability* and validity. 
Also as Suchman (1967) points out, like basic research, 
evaluative research requires a theoretical foundation. 
He goes on to stress that !* . . . a test of Does it work? 
presupposes some theory as to why one might expect it to 
work." (p. 86) 

There are at least three basic models of evaluative 
research which readily apply to state rehabilitation 
agencies : 

Model 1 - Measures the effect of an entire program. 
Example Research Question ~ What was the total 
gain in earnings of all clients rehabilitated 
by our agency? 

Model 2 - Compares the effect of specific variables 

within a program. 

Example Research Question - Do the clients of 
counselors with strong interpersonal skills 
have greater gains in earnings than the clients 
of counselors with weak interpersonal skills? 

Model 3 - Measures the differential effects of specific 

variables within a program. 

Example Research Question - Does the effect of 
counselor interpersonal skill level on clients 
depend on the type of client served? 



2 XO 



Although all three models could be used for evalua- 
tion purposes, Model 3* is seen as having the greatest 
utility for providing data on which state rehabilitation 
agency administrators can make effective program decisions. 
This is the result of two factors. First, there are many 
different types of clients (old, young; physically dis- 
abled, emotionally disabled, intellectually impaired; etc.) 
with differing needs seeking rehabilitation services at 
state rehabilitation agencies. Second, there are many 
different types of service programs available. Conse- 
quently it could be hypothesized that Model 3 effectively 
and comprehensively implemented could throw light on the 
differential effectiveness of service programs for re- 
habilitating client types with different needs. It can 
easily be assumed that such data would be invaluable to 
agency decision makers. 

At this point a fictitious example of Model 3 in 
operation will be utilized to demonstrate its application. 
The research question is: 

* ** 

Are comprehensive rehabilitation facilities 
differentially effective in their attempt to 
rehabilitate physically disabled and psychia- 
trically disabled clients? 



*Por a more elaborate discussion of this model see 
Bolton and Rubin (1974). 



3 11 



Table 1 lists the variables (independent, moderatpr, and 
dependent) . Table 2 illustrates Model 3 in a 2 X 2 two-way 

TABLE 1 
Project Variables 



Independent 

Facility 

Variables 



Moderator 

Client 

Variables 



Dependent Client 

Outcome 

Variables 



Facility A 



Facility B 



Physically 
Disabled 



Psychiatri- 

cally 

Disabled 



Change in earnings- 
from intake to 
closure 



TABLE 2 

Two-Way ANOVA Design for Studying 
Facility-Client Interaction 



Independent Client Moderator Variables 

Variables 

Physically Psychiatrically 
Disabled Disabled 



Facility A 



Facility B 



(Cell 1) 
Client change 
in earnings 

(Cell 3) 
Client change 
in earnings 



(Cell 2) 
Client change 
in earnings 

(Cell 4) 
Client change 
in earnings 
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analysis of variance. The six possible outcomes which 
can result from this design are listed in Table 3. 



TABLE 3 



Possible Outcomes 



1. No significant 
difference 



4. Facility and client 
main effect without 
interaction effects 



2. Facility main 
effect 



5. Ordinal interaction 
effect 



3. Client main effect 



6. Disordinal interaction 
effect 



Only two of these outcomes, ordinal interaction effect 
and disordinal interaction effect, require considera- 
tion of the two sets of variables simultaneously. 

The ordinal interaction effect is diagrammed in 
Table 4 . An ordinal interaction effect indicates that 
working with one type of client, one facility produces 
results superior to another, whereas, the extent of 
change for a second type of client is much less related 
to the facility entered. For example. Facility A may be 
significantly more effective than Facility B v;ith psychia- 
trically disabled clients, although they may show little 
difference in effectiveness when working with physically 
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disabled clients. If such a finding were real it would 
obviously suggest that most psychiatrically disabled 
clients should be sent to Facility A. 



TABLE 4 



The Effect of Two Facilities on Types of Clients 





Facility 


Client 




Client 


Variable 


' Moderator 




Outcome 




Psychiatri- 




$60 average 




cally 


produces 


increase in 




Disabled 




earnings 


Facility A 








working 








with: 










Physically 
Disabled 


produces 


$70 average 
increase in 
earnings 




Psychiatri- 




$40 average 




cally 


produces; 


increase in 


Facility B 


Disabled 




earnings 








working 








with: 










Physically 
Disabled 


produces 


$70 average 
increase in 
earnings 


A disordinal interaction 


effect indicates that one 


facility is 


more effective with one type 


of client while 



another facility is more effective with another type of 
client. The disordinal interaction effect is diagrammed 



in Table 5. Obviously, if such a finding were real it 
would suggest that psychiatrically disabled clients should 
be sent to Facility A while physically disabled clients 
should be sent to Facility B* 



TABLE 5 

The Effect of Two Facilities on Two Types of Clients 



Facility 
Variable 


Client 
Moderator 




Client 
Outcome 


Facility A 

working 

with: 


Psychiatri- 
cally 
Disabled 


produces 


$70 average 
increase in 
earnings 




Physically 
Disabled 


produces 


$60 average 
increase in 
earnings 


Facility B 

working 

with: 


Psychiatri- 
cally 
Disabled 


produces 


$30 average 
increase in 
earnings 




Physically 
Disabled 


produces 


$80 average 
increase in 
earnings 



Assuming research competence, the effective utiliza- 
tion of Model 3 or any other complex research design is 
dependent on accessibility to data on program activity. 
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client characteristics, and client outcome. This requires 
that the evaluation researcher have the cooperation of 
state rehabilitation agency administrative and service 
personnel. Such cooperation cannot automatically be 
assumed because of at least two factors: 

1. Fear that the research results will have a 
negative effect on program and personnel as 
opposed to a positive effect. 

2. Resistance to the additional work that will 
be required to provide data to researchers. 

Such impediments to effective evaluation research can 
probably be greatly overcome by providing many opportuni- 
ties to state rehabilitation agency administrative and 
service personnel to participate in the development of 
the research questions as well as the research plan. 
Such participation should also greatly increase the like- 
lihood that the research results will lend themselves 
to utilization as well as be utilized. 
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PROBLEiMS IiNf MEASURING OUTCOMES 
IN REHABILITATION RESEARCH PROJECTS* 

Brian Bolton 

Arkansas Rehabilitation Research and Training Center 
University of Arkansas 

In troduction; The Criterion Proble m 

Measures of client outcome have traditionally been 
utilized for two purposes: (1) as a basis for more fairly 
assessing counselor performance through the determination 
of caseload difficulty, and (2) as a basis for assessing 
agency effectiveness through the determination of client 
benefit. Underlying the rehabilitation philosophy is the 
assumption that successful vocational adjustment is funda- 
mental to satisfactory adjustment in other areas of life. 
Therefore, client benefit has traditionally been synonymous 
with vocational benefit. In the last few years some reha- 
bilitators have questioned the applicability of this as- 
sumption (e.g., Athlestan, et al. , 1973). Thus, emphasis 
has been placed on the assessment of personals-social adjust- 
ment as a worthwhile goal independent of vocational success. 
Issues in the Measurement of Outcomes 

Four conceptual issues which are relevant to the mea- 
surement of client outcomes have been delineated by the 
author in the form of dichotomies (Bolton, 1974a) : 



*This paper is based on a presentation at the American 
Personnel and Guidance Association Convention in New Orleans, 
April 11, 1974. 
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1. Economic (vocational) versus non-economic 
(personal adjustment) measures of outcome. 

2. Unidimensional (single score) versus multi- 
dimensional (multi-score) measures • 

3. Absolute measures (assessment of final status 
irrespective of initial status) versus rela- 
tive measures* 

4. Self- report versus observer rating measures 
(see Fairweather, et al. study presented 
later) • 

As can readily be seen the above four dichotomies re- 
sult in 16 possible combinations of criterion characteristics 
For example, the combination of "economic-absolute-unidimen- 
sional-objective" is illustrated by salary or Vocational 
Adequacy at Closure (see below) and the "noneconomic-relative 
multidimensional-self report*^ combination is illustrated by 
personal adjustment change measures. 

The standard procedure followed in most rehabilitation • 
research projects has been to enumerate a variety of rele- 
vant criteria and conduct separate analyses for each one 
(the "shotgun" approach) . This multiple criteria approach 
is distinguished from a multidimensional measure of reha- 
bilitation outcome in that the latter involves a systematic 
design and statistical procedure. 
Selected Measures of Client Outcome 

Several measures of client outcome have been developed 
in recent years in response to the criterion problem. They 
can be grouped into three unidimensional and four multi- 
dimensional instruments. The unidimensional instruments 



are reviewed below. 

1* Successful versus unsuccessful closure (R-300) ; 
the dominant measure of rehabilitation effective- 
ness (inadequate) • 

2. Vocational Adequacy at Closure ; a weighted com- 
posite score developed by Ebef (1966) which in- 
cludes work status, job code, weekly earnings, 
and closure status (economic) . 

3. Rehabilitation Gain : a composite score developed 
by Reagles, et al. (1970) which is based on 20 
items which reflect vocational success and per- 
sonal-social adjustment. 

The above three unidimensional measures of client out- 
come possess the advantages of comprehensiveness and ade- 
quate reliability. Their weakness is that they may be 
averaging several meaningful independent criteria into a 
combination score whicn has no specific interpretation 
(i.e.. Can vocational success and personal-social adjust- 
ment be meaningfully added together and labelled gain? 
Is it possible for clients to succeed vocationally and 
adjust poorly in other areas, and vice versa? If so, wnat 
does their gain score represent?) . The four multidimen- 
sional measures listed below were developed to assess 
several relatively independent dimensions of client outcome. 

4- Human Service Scale ; a self-report questionnaire 
developed by Reagles, et al. (1973) which results 
in scores on seven "need" factors, e.g., physio- 
logical needs, emotional-security needs, etc. 

5* Rehabilitation Services Outcome Measure ; a coun- 
selor rating instrument developed by Westerheide 
and Lenhart (1973) whicn emphasizes the client's 
functional ability in relation to employment; six 
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subscale scores result.: difficulty, educational 
status, economic/vocational status, etc. 

6. PVR Project Personal Adjustment Change ; a proce- 
dure which scores the pre/post responses of clients 
to the TSC, 16 PF, and Mini-Mult on six dimensions 
of change: self -concept, anxiety, extroversion, 
etc. (Rui?in, et al. , 1974). 

7. PVR Project Vocational Cnange : a procedure which 
scores the pre/post R-300 variables of salary and 
work status on three (highly correlated) measures 
of change: salary change, work status change, 
and vocational success (Krauf t, et al. , 1974) . 

Measurement of Change 

The quantification of change continues to constitute 
a major problem for psychometricians and applied researchers 
alike. Two major procedures, or some variants thereof, are 
in standard use. The first, the raw change score, is simply 
the difference between posttest and pretest performance. 
The raw change score procedure is inappropriate when ceil- 
ing effects or regression effects are serious problems. 
The residual change score may be a useful substitute under 
the conditions present wnen the raw score is inappropriate. 
Essentially, the residual change score compares individuals 
who have the same pretest status. O'Conner (1972) has 
provided the most up-to-date summary of the issues in 
measuring psychometric change. 
Pimensionality of the Criterion 

As suggested by the multidimensional measures outlined 
above, rehabilitation outcome is probably a multidimensional 
construct. Three research studies which further support 
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this conclusion are summarized below: 

1. Pairweather, et al. {I960) intercorrelated 13 cri- 
terion measures collected in conjunction with a 
research project studying hospitalized psychiatric 
patients. Two relatively independent clusters ap- 
peared: (a) subjective self-evaluation via self- 
report inventories, and (b) objective interpersonal 
evaluation reflected in follow-up items and group 
tnerapy ratings. 

2. Rosen, et al. (1970) factor analyzed 22 criteria 
of community adjustment for a sample of mentally 
retarded clients. Five factors resulted; coping 
behavior, job satisfaction (A), job stability, 
job satisfaction (B) , and general adjustment. 

3. Bolton (1974b) factor analyzed 12 measures of change 
in psychological adjustment (self-report) and three 
measures of change in vocational adjustment (R-300) . 
Three factors resulted: I. Improvement in work 
status and salary, II. Improvement in self-concept 
(or Reduction in neurosis) , and III. Reduction in 
psychosis. The first factor was uncorrelated with 
the other two factors. Thus, improved vocational 
functioning may be independent of improved psycho- 
logical adjustment. 

The Role of Factor Analysis in the Development 
of Rehabilitation Treatment Models 

The development of quantitative models of the reha- 
bilitation process constitutes an important step toward 
understanding how client characteristics interact with 
treatment parameters to produce successful outcomes. Such 
models can be effectively utilized to sort multiple vari- 
ables into meaningful categories under the client and 
outcome components of any rehabilitation treatment model. 
They are less appropriate for treatment variables which 
are usually pre-determined by practitioners who must 
operate under restrictions imposed by realistic 



14 



22 



considerations such as cost and client-staff ratio. Gen- 
erally speaking, the treatment variables are not reducible 
or otherwise subject to statistical re-combination. The 
three major components of any rehabilitation treatnient 
model may be further identified as follows: 

A. Client variables 

1. Attributes (relatively fiked measures such 
as sex, age, education, etc.) 

2. Characteristics (relatively modifiable or 
changeable dimensions such as motivation, 
self-concept, vocational skills, etc.) 

3. Treatment variables 

1. Counselor interpersonal skills and coun- 
seling techniques (friendliness, advising, 
occupational information, etc.) 

2. Services (physical restoration, psycho- 
therapy, personal adjustment training, etc.) 

C* Outcome variables 

1. Measured change on psychological charac- 
teristics (by self-report or observer rating 
techniques) . 

2. Global indices of success (program comple- 
tion, employment, self-sufficiency, etc.). 

Some outcome variables may be reduced to composite 
dimensions while others should nor be. In order to report 
the results of rehabilitation treatment efforts, adminis- 
trators are required to cite statistics which reflect 
directly upon economic issues, e.g., employment, welfare 
recipients, etc. Thus, theoretical models of the reha- 
bilitation process should be designed to incorporate 
directly the measures of treatment effort and outcome. 

The situation is entirely different in the domain of 
client variables. Except for the need to record basic 
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attributes in order to assign clients to optimal treat- 
ments, client variables should be reduced to more funda- 
mental dimensions whenever possible. Factor analysis is 
used to achieve parsimony in the client domain, which 
usually contains the largest number of variables. The 
factors then represent the major theoretical constructs 
in the psychological model of the rehabilitation process. 
The model gains generalizability by using the stable di- 
mensions of client functioning underlying the treatment 
process. 

Treatment Models 

In the final section of this paper two basic rehabili- 
tation treatment models are outlined: 

Model I . This model assumes that the client's initial 
status regarding psychological characteristics in the do- 
main of abilities and skills and intensity of services 
interact to determine the rehabilitation outcome. Factor 
analysis is employed to reduce the client initial status 
characteristics (which are assumed to be relatively stable) 
to fundamental dimensions. A moderated multiple regression 
analysis or computer pattern analysis (AID) is used to 
assemble the client, treatment, and outcome variables. 

Model II . This model is appropriate for studying the 
counselor as the primary effector of change in the client. 
The assumption is made that various combinations of 
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counselor characteristics and skills interact with client 
characteristics in producing client change and rehabili- 
tation success. Or stated in a slightly different manner, 
the effect of the counselor is moderated by client charac- 
teristics. The dependent variables in this ANOVA design 
can be psychometric change' scores or global indices of 
rehabilitation success. Factor analysis is employed to 
isolate the primary dimensions of client change. 
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ASSESSMENT OP EXTERNAL EFFECTS 
IN REHABILITATION RESEARCH* 



Reed Greenwood, Ed.D, 
Arkansas Rehabilitation Research and Training Center 
University of Arkansas 

Research, particularly of an evaluative type, has 
recently become a matter of central focus in the field 
of rehabilitation. The emphasis from the present exec- 
utive and congressional leadership, is for program ac- 
countability and demonstrated program effectiveness. In 
the past, vocational rehabilitation programs have often 
been supported and funded because of noteworthy intent 
and a deserving client population, in addition to dem- 
onstrated and assiimed effectiveness. However, the era 
of reliance upon input measures of effectiveness and ' 
gross, cumulative output measures (rehabilitation clo- 
sures) is bygone. Present policies not only encourage, 
but mandate more sophisticated evaluations of public 
vocational rehabilitation programs. This mandate places 
a heavy responsibility upon program administrators who 
in turn look to the expertise of rehabilitation re- 
searchers for guidance in developing program evaluation 
techniques . 



*This paper is based on a presentation at the American 
Personnel and Guidance Association Convention in New 
Orleans, April 11, 1974. 
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Researchers do have an arsenal of techniques and 
evaluation methods available for use in assessing the 
effectiveness of rehabilitation efforts. However, these 
techniques are more often appropriate for controlled ex- 
perimental conditions *rather than for applied research 
where tightly controlled designs may not be possible. The 
challenge of identifying evaluation methods suitable for 
utilization with ongoing rehabilitation programs is 
considerable. The purpose of the following discussion 
is to focus on a somewhat neglected area of concern in 
which the need for careful planning and conceptualization 
for evaluation is essential; namely, the influence of 
the environment extei^nal to the rehabilitation process. 
Although external factors are difficult to assess in 
terms of strength of effect, they may contribute con- 
siderably to rehabilitation outcome. 

Because the primary focus of the rehabilitation pro- 
cess has been the rehabilitation client , heavy emphasis 
in rehabilitation research has been placed on the indi- 
vidual and his effectiveness in coping with the environ- 
ment. Efforts to predict rehabilitation outcome using 
client characteristics have resulted in findings which 
could account for only a portion of the 'variability con- 
tributing to outcome. After reviewing 40 such prediction 
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studies, Bolton (1972) found few which accounted for more 
than 25 percent of the variance in outcome. The remain- 
ing variance is determined by such factors as treatment 
parameters, client-counselor interaction, and other fac- 
tors external to the rehabilitation process. These ex- 
ternal factors include, but are not limited to, economic 
conditions, institutional barriers, and societal at- 
titudes which may differentially affect the disabled. 

Before looking more closely at these specific ex- 
ternal factors, a framework for model building and 
analysis is needed. Mullen and Dumpson (1972) offer just 
such a framework for evaluative research in programs 
like vocational rehabilitation. Drawing upon general 
systems theory, three levels of systems are proposed: 
(1) microsystems , in which problems are viewed in terms 
of the determining focus being within the boundaries of 
either an individual or a small group directly experi- 
encing the problem, (2) mez2sosy stems , in which the de- 
termining forces are seen as directly involving not only 
human systems but social systems on the scale of neigh- 
borhood and communities, and (3) macrosy stems , in which 
the critical determinants extend beyond given individuals, 
groups, or localities, and occur on the broadest level 
of social organization involving large geographically 
scattered populations. Having (Emphasized microsystem 
levels in the past, rehabilitation researchers are now 
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required to move to mezzo-and macrosystem level analysis 
for a more adequate understanding of the. multiple variables 
affecting rehabilitation outcome. Evaluation research 
models by necessity must be developed for these various 
levels if additional understanding and knowledge are to 
be obtained. 

Fairweather (1967) offers some guidance regarding the 
specific external factors which should be incorporated 
into a more comprehensive model. The more potent of his 
suggestions are social climate, geographical location, 
social mores and folkways, and economic conditions. For 
disabled populations it would also be necessary to in- 
clude institutional policies and practices. 

The identification of various external factors in- 
fluencing the evaluation research process and the measure- 
ment of their effect on outcome is not an easy task. How- 
ever, examining the mezzo- and macrosystem factors in more 
detail may result in some useful, if crude, measures. 
Social Climate; Multiple factors are involved in 
assessing the social climate in which a social ser- 
vice system such as vocational rehabilitation func- 
tions. However, as an example of assessment of one 
aspect of social climate, the degree of acceptance 
of the rehabilitation process and system can be as- 
sessed at both mezzo- and macrosystem levels. Through 
carefully conducted surveys, levels of acceptance by 
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various groups can be assessed. This would provide 
additional insight into the problems and obstacles 
to be encountered in communities and larger geograph- 
ical areas, if a generally supportive social climate 
exists, the realization of positive rehabilitation 
outcomes is obviously enhanced. Other areas for fur- 
ther research and study include attitudes toward the 
disabled (particularly attitudes relating to employ- 
ability) , attitudes toward roles such as husband/wife, 
cultural and religious biases, and the sympathy- 
apathy-charity public response triad. 
Geographical Location ; Obviously attention must be 
given to the similarities and differences which exist 
among and between rural-urban, county, state, city, 
or national areas. The effect of geography including 
climate, topography, and other variables has been 
recognized in cultural^ social, and economic devel- 
opment. It is also an external factor which should 
be identified and assessed in terms of its effect upon 
rehabilitation outcome. 

Social Mores and Folkways ; The variability among 
subsystems within a larger geographical area in terms 
•of social mores and folkways is considerable and 
could account for ,some outcome variability. For 
example, the mores of simple hill folk in regard to 
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the disabled may vary greatly from those of more 
sophisticated urban dwellers. This variability has 
often been lightly dismissed because of the assumed 
randomization of such factors across the population. 
However, it is an external variable which requires 
greater attention, as well as a more comprehensive 
assessment of the effect such mores have on reha- 
bilitation outcome. 

Economic Conditions ; This is one of the more po- 
tent external factors, and one for which a consid- 
erable amount of data exist at both mezzo- and 
macrosystem levels. Attention should be directed 
toward assessing the effects of employment (or un- 
employment) rates, technological change, the rate 
of economic development (or underdevelopment) , and 
the production/profit vs. value of the individual 
attitudes of employers. One rather obvious problem 
is that the differential effect of unemployment on 
minority groups such as Blacks would likely apply 
to the disabled, thus creating placement difficulties 
Close scrutiny of economic indicators and the effect 
of such indicators on rehabilitation outcome is 
crucial to the evaluation process. 

Institutional Policies and Practices ; The complexitii 
of existing private and public institutions have been 
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recognized as influencing the success of the rehabili- 
tation process. However, the effect of such "competing" 
public policies as social security disability insur- 
ance and vocational rehabilitation programs has not 
been clearly documented. Although complementary in 
many ways, vocational rehabilitation and social se- 
curity programs offer competing opportunities (voca- 
tional rehabilitation leading to. independence on one 
hand versus social security emphasizing financial 
dependency on the other) . These policies have a wide- 
spread macrosystem impact upon rehabilitation out- 
come since they exist nationwide. 

In summary, several issues have been reviewed for con- 
sideration in conducting program evaluation research. 
These were: 

(1) Rehabilitation outcome is determined by a complex 
group of factors including participating clients, 
treatment staff and services (the rehabilitation system) , 
and factors external to the system. 

(2) Microsystem research on the client or small groups 
has resulted in findings which account for only a 
limited portion of the variability in rehabilitation 
outcome. 

(3) A systems approach will provide a better model 
for conceptualizing the boundaries of prograra evalua- 
tion research and for expanding those boundaries to 
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mezzo-and macrosystem levels • 

(4) At mezzo- and macrosystem levels the identifica- 
tion of factors external to the rehabilitation system 
and the measurement of such factors should and can 

be realized^ 

(5) Specific external factors were identified and 
discussed. These were limited to social climate, geo- 
graphical location, social mores and folkways, economic 
conditions, and institutional policies and practices. 
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ISSUES IN INITIATING LARGE SCALE 
REHABILITATION RESEARCH* 

Richard Roessler, Ph.D. 
Arkansas Rehabilitation Research and Training Center 
University of Arkansas 

Introduction 

The purpose of this paper is to describe problems 
encountered in conducting large scale rehabilitation re- 
search studies. In particular, coitunents in the paper 
refer to the Services Coordination Project currently un- 
derway at the Arkansas Services Center in, Jonesboro, 
Arkansas. The Arkansas Rehabilitation Research and Train- 
ing Center has contracted with Social and Rehabilitative 
Services of Arkansas to evaluate the coordination project. 

In developing and administering the evaluation of 
the project, research staff have identified a number of 
social and political forces that affect the conduct of the 
research. The remainder of the paper attempts to describe 
these areas of conflict and to recommend methods of resolution* 

Specifying Objectives 

The first place that evalu^ftion research interacts 
with the real world is in the setting of objectives and 
goals for the social impact program. 



*This paper is based on a presentation at the American 
Personnel and Guidance Association Convention in New 
Orleans, April 11, 1974. 
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As Weiss (1973) points out, projects often have diffi- 
culty translating Federal legislation into program activi- 
ties because legislation specifies neither the operations 
involved in reaching goals nor the evaluative criteria for 
determining whether goals have been reached. The legis- 
lative base for the 'Services Coordination Project, the 
Allied Services Actl, itself does not clarify such terms 
as coordination, independence, dignity, and, to a lesser 
degree, self-sufficiency. 

Compared to What? 

Unclear objectives in legislation create an interesting 
projective test for State, project and evaluation personnel. 
In terms uf Services Coordination, state and project per- 
sonnel saw the charge of the Allied Services Act as one 
of defining and implementing a management system to coor- 
dinate social and rehabilitative services. 

The evaluation staff, on the other hand, was concerned 
with questions regarding effectiveness and efficiency, 
e.g., is the experimental service program superior to the 
traditional one? In other words, to determine if services 
coordination is an improvement, one must ask, "Compared 
to what?"; obviously, "Compared to nonintegrated service 
delivery. " 



^Presently under consideration in the Senate. 
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Specifying ways in which services coordination should 
be an improvement over nonintegrated services involves 
defining social change criteria (Fairweather , 1971) . 
According to Pairweather, social change criteria represent 
a consensus among interested parties regarding what prob- 
lems the program should solve. 

Since such consensus rarely exists, evaluators end 
up initiating a process that should have started with 
Federal legislation, that is, developing appropriate com- 
parisons in terms of relevant social change criteria. 

Asking such questions as "Compared to what?" does 
not endear rese^-rchers to those who have the responsibility 
for initiating and maintaining a system. Indeed, for 
project maintenance purposes, the only real need for evalu- 
ation is to answer formative or management oriented ques- 
tions. For example, in the Services Coordination project. 
State and project staff wanted the evaluation team to 
compare alternative management procedures for carrying out 
coordinated service delivery. 

Evaluation personnel, however, felt that the first 
priority was to demonstrate a need for change in current 
practices. Establishing the need involved those effec- 
tiveness and efficiency questions, which require system 
comparisons in terms of client outcome and case process. 

Due to differing perceptions of evaluation, dis- 
cussion went back and forth within the project as to the 



best way to proceed with the evaluation- Resolution of 
tne question of management versus outcome evaluation 
came during a joint meeting with Federal, State, project, 
and evaluation personnel. :?he balance was tipped when 
Federal project officers emphasized the importance of the 
effectiveness and efficiency issues; Federal representa- 
tives were less interested in "fine tuning" and more in- 
terested in comparative outcome data. 

At first, the time invested in the dialogue between 
tne evaluation staff and project and State personnel 
seemed wasted effort. But, as a result of the dialogue, 
tne evaluation team recognized the need for a balanced 
evaluation study. The final evaluation design for Ser- 
vices Coordination, as a result, included studying client 
outcome in coordinated and noncoordinated service delivery 
and studying the differential effects of alternative 
management procedures to carry out case handling in ser- 
vices coordination. 

Relationship of Evaluation Process to Project 

Differing perceptions of evaluation illustrate the 
need for greater communication and coordination among 
Federal, State, project, and evaluation representatives. 
However, such communication raises a question regarding 
the relationship of evaluation to the overall project. 
In the past, evaluation literature has stressed the need 
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for objectivity; the evaluation team should avoid contacts 
that might contaminate its objectivity. If necessary, 
the team should speculate regarding program objectives 
rather than communicate directly with project staff 
(Freeman and Sherwood, 1965) . 

Though objectivity is not overlooked, current trends 
emphasize impact and utilization of evaluation, which 
necessitates a relationship between evaluation and project 
efforts. Indeed, Freeman and Sherwood (1965) contend 
that the evaluator is socially responsible for working with 
social impact prograi..^ 1 the development of measurable 
objectives and appropriate program practices. 

Experimental Treatment: Definition and Acceptance 

Involvement of evaluation personnel or appropriate 
consultants in the development of project objectives and 
activities helps overcome another problem often cited in 
evaluation research, that of a constantly changing experi- 
mental program. If the program is constantly changing, 
it becomes increasingly difficult to determine the nature 
of the experimental treatment; i.e., to identify program 
features that might have had a positive effect. 

Initial planning for the Services Coordination Project 
involved extensive use of a management consultant to de- 
velop an operational management system. The consultant 
did his job well and, from the viewpoint of the evaluation 
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team, developed a system, that could, in fact, coordinate 
tne delivery of social services. A clear experimental 
treatment had been designed. 

The irony of the project was that the operational 
management system was slow to move into full operation. 
The delay seemed to result from (a) the perceived threat 
of coordinated services, (b) the lack of perceived need 
for services coordination, (c) the lack of involvement of 
service delivery people in the planning of the managemt <t 
system, and (d) the lack of authority of the project over 
service delivery personnel. 

The time spent in defining project activities and 
^in implementing those activities has a tremendous impact 
on evaluation. For research on Services Coordination, 
the chief loss was in terms of data collection time for 
pre- and post-testing. 

With better planning, the time lag-^might have had 
some value to the project. For example, the period be- 
tween program development and program implementation 
could have been used to gather baseline data regarding 
current client outcomes in noncoordinated services. 

However, eagerness to start the project resulted in 
tentative coordination efforts that contaminated the 
setting. Service providers realized ^that traditional 
service delivery was no longer in existence, although they 
were not always sure what had replaced it. 
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Eagerness to begin is certainly a positive attribute 
of any project. But, it, too, can be affected by some of 
the same political and social forces that affect evaluation • 
For example, the Services Coordination project has been 
hampered^ by perceived shifts in commitment on the Federal 
level to coordination of service delivery. Such shifts 
are manifested in the on again/off again funding rumors 
that have a definite effect on project morale. 

Summary 

In summary, numerous factors, beyond those technical 
ones of research methodology, impinge on evaluation re- 
search. In terms of the Services Coordination project, 
these factors manifested themselves in the following 
problems: 

1. The problem of starting with objectives stated 
in political or, at least, nonoperational terms. 

2. The problem of multiple interpretation of 
ambiguous ob j ectives . 

3. The lack of clearly defined social change criteria. 

4. The conflicting expectations of evaluation for 
either management (formative) or outcome (summa- 
tive) studies . 

5. The problem of evaluation meeting its social re- 
sponsibility while at the same time maintaining 
its objectivity. 
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6. The problem of a constantly ^hanging treatment 
effect that makes it difficult to identify causes 
of program outcomes. 

7. The problem of program acceptance by service 
providers. 

8. The problem of project eagerness as it effects 
the collection of baseline data. 

9. The effect of changing political trends on pro- 
gram support and^ consequently, on morale of pror 
ject personnel. 

Responding to problems of research in an applied 
setting requires reviewing aspects ranging from Federal 
legislation to project enactment. Some recommendations 
for evaluation in social/political settings are: 

1. Evaluation specialists should participate in 
drafting Federal legislation to accomplish 
the following : 

a. Specify objectives in operational terms. 

b. Identify relevant social change criteria. 

c. Establish procedures to ensure that all 
proposals include appropriate objectives 
and criteria. 

d. Develop review methods for projects so 
that long-term funding can be committed 
contingent on specific project activities 
and behaviors . 
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2. Joint objectives writing sessions involving 
grantees (State and local) and intended or out- 
side evaluation personnel would help individual 
programs establish their goals in measurable 
terms . 

3. In establishing an evaluation program mutual dis- 
cussion among Federal, State, project, and evalu- 
ation personnel is needed regarding: 

a. Evaluation's role an'd responsibility 
in the project. 

b. The nature of social change criteria 
for the study and types of acceptable 
evidence for project success. 

4. Service providers need to be involved, in the de- 
velopment of the project in terms of stating 
objectives and developing program practices to 
meet those objectives. Involvement will build 
understanding of the need for the program and 
commitment for tarrying out program policies. 

5. Evaluators need to develop balanced evaluation 
studies including both outcome and management 
data, for example > comparing an innovation with 
present practice and experimenting with additional 
ways to implement innovative practices. 
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6. Baseline data should be collected before in- 
stituting innovative practices. Outcomes of 
ongoing programs can then be compared with 
those resulting from new programs. 
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REACTIONS ON EVi\LUATION* 



Jerold D. Bozarth, Ph.D. 
Iowa Mental Health Authority 
Department of Psychiatry 
University of Iowa 



One thing which evaluators do very often is to as-- 
svune that everyone knows the difference between evaluation 
and research. The difference between definitions of 
"evaluation" and "research" often cited are a) research 
is directed toward adding new knowledge while evaluation 
is directed toward noting information associated with a 
particular program; b) research utilizes high levels of 
"good" design, including randomization, control groups, 
etc., while evaluation is usually unable to establish 
such designs; and c) research is longitudinal and evalua- 
tion is short-term. Many of these differences do not, 
in fact, need to exist. In ^n attempt to throw light on 
both the state of art in evaluation research as well as 
examine the relationship between "research" and "evalua- 
tion", a number of points seem pertinent. These are: 

1) "Good" evaluation is the same as "good" research. 
That is, the evaluation research design should have good 



*This paper is based on a presentation at the American 
Personnel and Guidance Association Convention in New Orleans, 
April 11, 1974. 
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internal validity allowing confidence in the^ results and 
good external validity allowing for the practical appli- 
cation of the results within specified populations. Un- 
fortunately, most evaluation is at the lower end of the 
qualitative continuum in respect to validity considera- 
tions such as statistical design, control of variables, 
statistical power and .availability of cross-validation. 
Yet, it is quickly adapted for administrative actions; 
especially as a substantiation of administrative designs. 
Hence, evaluation is probably more harmful than helpful 
the way it is usually practiced. For example, one eval-* 
uation report that intelligence quotients could be im- 
proved by 100 points by altering environment was cited in 
a federal report — slightly misinterpreted — and latched 
onto by national and international media, producing strong 
incentives for legislators to act in certain ways. The 
lack of validity of the findings, as well as the misinter- 
pretation by some myopic bureaucrats had to be pointed out 
before there was even an awareness of great perpetuation 
of error • 

2) Evaluation research should be relevant. If 
evaluation research is to be relevent, three question 
areas should be considered. These questions are: 

a) Questions which are asked by legislators, boards 
of supervisors, the general public, etc. These 
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questions must often be reworded by the questioner, 
with the help of the evaluator. 

b) Questions which can assist particular programs to 
modify and revamp their effort towards more effect- 
iveness. 

c) Questions which bear on both a and b above and 
also on basic research to add new knowledge to 
questions of rehabilitation counselor effectiveness. 
This is basically information which can be applied 
to the broad question of "What kinds of counselors 
providing what kinds of services to what kinds of 
clients are being effective in what kinds of ways?" 

3) Evaluation generally has an accountability set which 
is, in the opinion of the present writer, a set which pro- 
duces only threat to, evasion of, game-playing-by , and re- 
sistance by the project personnel being evaluated. 

4) The data gathered via evaluation are seldom used, and 
often cannot answer the original questions. The questions 
which usually initiate evaluation components of programs 

are generally value-type questions, rather than measurable- 
type questions. in addition, evaluation research efforts, 
by-and-large, lead to a collection of invalid,, superfluous 
data which are seldom related to the original questions 
anyway. A conunon response of at least a number of federally 
funded mental health centers which are required to turn 
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in numerous accountability data is^ firsts they don't know 
how the information is used; second, the reports which 
are produced are of no reasonable value; and thirds infor- 
mation about essential questions is nearly impossible to 
retrieve . 

This should not be interpreted as a plea to eliminate 
evaluation. It is a plea for a focus on certain con- 
siderations. These considerations are: 1) that "basic" 
research must be included in the evaluation of programs. 
It is the long-term cross-validation and replication 
which is the crux of advances in rehabilitation programs; 
e.g., research in the pecking of pigeons (operant con- 
ditioning) eventually led to behavior modification ap- 
proaches to help people. Likewise ^ extensive research 
on variables related to effective psychotherapy has led 
to training programs for rehabilitation personnel, men- 
tal health center personnel, nurses, etc.; 2) that ques- 
tions originally posed by legislators, administrators^ 
etc.^ need to be altered so that they are posed in ways 
which lend themselves to m^|^^ent. Like doctoral dis- 
sertations, one of the maj^Hj^lems is finding and 
stating the questions. The remainder of activity to 
follow an adequately stated question is simply a complex 
technological footnote; and 3) that evaluation ought to 
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be a "helping" set, rather than determination of the worth 
(funding?) of a program. The set ought 'to be one which 
assists programs to identify their weaknesses and strengths 
and modify accordingly. An example of this approach oc- 
curred in a small, unpublished study which took place in 
a mental health treatment facility. Some measures of 
client goal accomplishment were developed, and implemented 
as one criterion. Over the span of a year/ it was found 
that better than sixty percent of the clients served de- 
picted zero improvement on the criterion measure. With 
an accountability set, this finding would have been quite 
threatening ( and resisted on the basis of being a gross 
study, having only one criterion, using poor evaluation 
procedures, etc.)* However, part of the evaluation in- 
cluded a more thorough examination of preliminary find- 
ings and was geared toward helping the center improve 
services. The further examination indicated that, on the 
one hand, a high percentage (90%) of clients of some of 
the therapists were improving, while a small percent 
(like zero) of the clients of other therapists were im- 
proving. On the other hand, it was found that clients 
receiving certain services were not improving while other 
clients receiving other services were split in their im- 
provement, depending upon the therapist with whom they 
had contact. This was an interesting finding, in that it 
corresponded closely to extensive reports of differential 
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effectiveness in psychotherapy literature. It was also 
a useful administrative bit of information, largely be- 
cause of -a "helping" set rather than an "accountability" 
set. The administration of the center was able to take 
the findings of clients' lack of improvement to the in- 
dividual therapists confidentially, with the idea of 
looking further at this phenomenon with the therapists. 
It was realized that this was only one criterion, but 
Still worth consideration by therapists^ since clients of 
some of the other therapists were improving on that same 
criterion. Such determinations were eventually established 
as that of one therapist changing assignments and feeling 
relieved about the change • Another therapist was able 
to change clientele and begin working with a less threat- 
ening clientele • The situation had cast the therapist 
into a necessary work load of clientele which posed 
problems for that therapist. The therapists and the cen- 
ter eventually benefitted from the initially threatening 
(in an accountability set) evaluation finding. 

The administration also identified, at least, one of 
the services as being inadequately structured. This was 
the service of group therapy, which the evaluation had 
revealed had few clients improving. The apparent reason 
was that there was no core group in the therapy sessions; 
that the clients might come for one or two weeks and never 
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return, and then a complete cycle of clients participated 
in the groups in this way. Alterations to include more 
core structure to the groups was thought to be a way to 
alter the apparent ineffectiveness of the group therapy 
program. 

All in all, evaluation is viewed by the present 
writer as being functionally weak and probably perpet- 
uating error. Evaluation, however, can be strong*.- It 
can be helpful* It can aid in the strengthening of 
programs . 
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